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SPACE  ACQUISITIONS 

DOD's  Goals  for  Resolving  Space  Based  Infrared 
System  Software  Problems  Are  Ambitious 


Why  GAO  Did  This  Study 

In  1996,  DOD  initiated  the  Space 
Based  Infrared  System  (SBIRS)  to 
replace  the  nation’s  current  missile 
detection  system,  and  to  provide 
expanded  missile  warning 
capability.  Since  then,  SBIRS  has 
been  restructured  several  times  to 
stem  cost  increases  and  schedule 
delays,  including  revising  program 
goals  in  2002,  2004,  and  2005.  These 
actions  were  partly  due  to  the 
challenges  of  developing 
sophisticated  technologies  and 
software.  In  2007,  SBIRS  had  a 
major  setback  when  flight  software 
for  the  first  satellite  underwent 
testing  and  failed,  a  failure  caused 
by  design  issues.  DOD  developed  a 
plan  for  resolving  these  issues,  and 
revised  its  cost  and  schedule  goals. 
GAO  has  assessed  (1)  the  approach 
used  to  mitigate  the  problems,  and 
(2)  the  cost  and  schedule  risks  and 
challenges  of  that  approach.  To 
conduct  our  work,  GAO  has 
contacted,  met  with,  and 
performed  detailed  work  at 
numerous  DOD  and  contractor 
offices;  and  reviewed  technical 
documents  on  flight  software. 


What  GAO  Recommends 


GAO  recommends  that  the 
Secretary  of  Defense  revise  cost 
and  schedule  goals  commensurate 
with  acceptable  risk  to  increase  the 
confidence  of  success,  and  require 
the  contractor  to  adhere  to 
disciplined  software  practices  as  a 
priority  to  reduce  risk.  DOD 
partially  concurred  with  the  first 
recommendation  to  revise  the  cost 
and  schedule  estimates,  and 
concurred  with  the 
recommendation  to  prioritize 
adherence  to  software  practices. 

To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  GAO-08-1073. 
For  more  information,  contact  Cristina  T. 
Chaplain  at  (202)  512-4841  or 
chaplainc  @  gao.gov. 


What  GAO  Found 

To  mitigate  the  SBIRS  flight  software  problems,  DOD  has  assessed  various 
alternatives  and  developed  a  way  to  implement  the  software  redesign  and 
oversee  its  development.  In  April  2008,  DOD  approved  the  redesign  effort, 
which  addressed  problems  with  the  original  design  that  affected  the  timing  of 
stored  programs,  distribution  of  control  between  processors,  and  failure  at  the 
hardware  interface  level.  Six  review  teams  comprised  of  70  personnel  in  all 
evaluated  the  designs  to  ensure  the  technical  solutions,  development 
approach,  and  readiness  of  test  facihties  were  adequate.  DOD  and  its 
contractor  are  now  implementing  the  simplified  architecture,  developing  new 
software,  and  testing  elements  critical  to  the  integration  and  test  of  systems. 
DOD  is  also  improving  its  program  oversight  and  better  managing  the  SBIRS 
development,  by  acting  on  the  recommendations  of  an  Independent  Program 
Assessment;  addressing  weaknesses  in  management  responsibility, 
accountability  and  organizational  structure;  and  estabhshing  a  central 
execution  team. 

DOD  has  estimated  that  the  SBIRS  program  will  be  delayed  by  15  months  and 
cost  $414  miUion  in  funding  to  resolve  the  flight  software  problems,  but  these 
estimates  appear  optimistic.  For  example,  confidence  levels — based  on  the 
program’s  ability  to  develop,  integrate,  and  test  software  in  time  to  meet  the 
schedule  goal — have  been  assessed  as  low. 


Confidence  Level  to  Produce  Software  in  Time  to  Meet  First  Satellite  Launch  Goal 

Confidence  level 

Contractors 

Estimated  launch  goal 

Less  than  10  percent 

Aerospace  Corporation 

December  2009 

5  percent 

Galorath,  Inc. 

December  2009 

50  percent 

Lockheed  Martin 

December  2009 

Source:  U.S.  Air  Force  (data);  GAO  (analysis  and  presentation). 


Further,  the  review  teams  who  approved  the  designs  to  start  coding  software 
report  that  the  program’s  aggressive  schedule  is  a  major  challenge  because  it 
allows  “little  margin  for  error.”  DOD  has  also  introduced  risk  by  granting 
waivers  to  streamline  the  software  development  processes  to  meet  the 
aggressive  schedule.  These  allow  the  program  to  deviate  from  disciplined 
processes  in  order  to  compress  the  schedule  and  meet  the  goal.  In  addition, 
some  software  elements  are  behind  schedule,  and  thousands  of  software 
activities  and  deliverables  remain  to  be  integrated.  Delay  by  these  other 
programs  could  create  unintended  consequences  for  the  SBIRS  launch  goal.  If 
DOD  should  need  additional  time  or  encounter  problems  beyond  what  was 
plarmed  for,  more  funds  will  be  needed  and  launch  of  the  first  satellite  in 
December  2009  could  be  jeopardized. 
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Abbreviations 


DOD 

Department  of  Defense 

FFRDC 

federally  funded  research  and  development  center 

GEO 

geosynchronous  earth  orbit 

HEO 

highly  elliptical  orbit 

IPA 

Independent  Program  Assessment 

OSD 

Office  of  the  Secretary  of  Defense 

RFSW 

reusable  flight  software 

SBIRS 

Space  Based  Infrared  System 

This  is  a  work  of  the  U.S.  government  and  is  not  subject  to  copyright  protection  in  the 
United  States.  The  published  product  may  be  reproduced  and  distributed  in  its  entirety 
without  further  permission  from  GAO.  However,  because  this  work  may  contain 
copyrighted  images  or  other  material,  permission  from  the  copyright  holder  may  be 
necessary  if  you  wish  to  reproduce  this  material  separately. 
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Accountability  *  Integrity  *  Reliability 


United  States  Government  Acconntability  Office 
Washington,  DC  20548 


September  30,  2008 
Congressional  Committees 

In  1996,  the  Department  of  Defense  (DOD)  initiated  the  Space  Based 
Infrared  System  (SBIRS),  a  satellite  missile  warning  system,  to  replace  the 
nation’s  current  missile  detection  system  and  to  provide  expanded 
capabilities  to  support  intelligence,  surveillance,  and  reconnaissance. 

Since  its  inception,  SBIRS  has  been  burdened  by  imderestimated  software 
and  technical  complexities,  poor  oversight,  and  other  problems  that  have 
resulted  in  cost  overruns  and  years  in  schedule  delays.  DOD  had  expected 
to  field  SBIRS  by  2004  at  a  cost  of  $4.2  bilhon;  however,  SBIRS  is  now 
estimated  to  cost  over  $10.4  billion,  and  the  first  satellite  launch  is 
expected  in  2009 — a  7-year  delay. 

In  2006,  you  requested  that  we  review  the  SBIRS  program.  In  response,  we 
reported  on  an  array  of  problems  the  program  was  still  facing,  particularly 
with  respect  to  software  development,  the  expenditure  of  management 
reserves,  and  deferred  requirements.^  Subsequent  to  our  work,  SBIRS 
experienced  another  major  setback  in  January  2007  when  the  flight 
software  for  the  first  satellite  underwent  testing  and  failed.  The  flight 
software  controls  and  monitors  the  satellite’s  health  and  status  and  is 
considered  a  critical  component  of  the  satellite.  In  April  2007,  DOD 
determined  that  the  software  failure  was  caused  by  design  issues  that 
affected  the  timing  of  stored  programs,  among  other  problems.  DOD  also 
developed  a  plan  for  resolving  the  issues,  and  associated  cost  and 
schedule  goals. 

Given  the  importance  of  flight  software  to  the  first  SBIRS  satellite  and  its 
cost  and  schedule  impact  on  the  SBIRS  program,  we  agreed  to  follow  up 
on  our  work  and  assess  the  software  management,  development,  and 
mitigation  efforts.  Specifically,  we  (1)  identified  DOD’s  approach  to 
mitigate  the  SBIRS  flight  software  problems,  and  (2)  assessed  the  cost  and 
schedule  risks  and  challenges  of  that  approach. 

To  conduct  our  work  for  this  report,  we  contacted  the  Office  of  the 
Secretary  of  Defense  (OSD),  Air  Force,  and  contractor  offices.  We  also 


^GAO,  Defense  Acquisitions:  Space  Based  Infrared  System  High  Program  and  its 
Alternative,  GAO-07-1088R  (Washington,  D.C.:  Sept.  12,  2007). 
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conducted  detailed  work  and  held  discussions  with  both  the  Air  Force  and 
Lockheed  Martin  on  their  efforts  to  manage,  mitigate,  and  redesign  the 
flight  software  that  is  to  operate,  control,  and  monitor  the  satellite’s 
health,  status,  and  safety.  We  reviewed  technical  software  plans, 
assessments,  analyses,  and  independent  reviews  pertaining  to  the  flight 
software’s  redesign,  and  held  discussions  with  key  Air  Force  and 
contractor  officials  on  various  aspects  of  the  flight  software  development 
for  SBIRS.  In  addition,  we  drew  from  our  body  of  past  work  on  weapon 
systems  acquisitions  practices  and  used  disciplined  software  practices  as 
criteria.^  We  conducted  this  performance  audit  from  April  2008  to  August 
2008  in  accordance  with  generally  accepted  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives.  Appendix  1  further  discusses  our  scope  and 
methodology. 


Results  in  Brief 


DOD  has  assessed  various  alternatives  for  mitigating  SBIRS’  flight 
software  problems  and  developed  a  way  forward  to  implement  the 
program’s  software  redesign  and  oversee  its  development.  In  April  2008, 
DOD  approved  the  overall  software  redesign  effort  which  was  to  address 
problems  with  the  original  design  that  affected  the  timing  of  stored 
programs,  distribution  of  control  between  processors,  and  failure  at  the 
hardware  interface  level.  Review  teams — comprised  of  personnel  from  the 
Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics;  Aerospace  Corporation;  Lockheed  Martin  Corporate;  Air  Force 
Space  and  Missiles  Systems  Center  Wing;  and  Software  Engineering 
Institute — evaluated  the  designs  to  ensure  the  technical  solutions, 
software  requirements,  development  approach,  and  readiness  of  the  test 
facilities  were  of  adequate  quality.  Currently,  DOD  and  the  contractor  are 


^CMMI®  (Capability  Maturity  Model®  Integration)  is  a  collection  of  best  practices  that  helps 
organizations  improve  their  processes.  It  was  initially  developed  by  product  teams  from 
industry,  government,  and  the  Software  Engineering  Institute  for  process  improvement  in 
the  development  of  products  and  services  covering  the  entire  product  life  cycle  from 
conceptualization  through  maintenance  and  disposal.  Following  the  success  of  CMMI 
models  for  development  organizations,  a  CMMI  model  that  addresses  the  acquisition 
environment  was  developed;  and  can  be  found  within  Guidelines  for  Successful 
Acquisition  and  Management  of  Software-Intensive  Systems:  Weapon  Systems 
Command  and  Control  Systems  Management  Information  Systems,  Department  of  the 
Air  Force,  Software  Technology  Support  Center,  (Condensed  version)  (February  2003). 
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working  to  implement  the  simplified  architecture,  develop  additional 
software,  and  test  elements  critical  to  the  integration  and  test  of  systems. 
DOD  has  also  undertaken  several  initiatives  to  improve  its  program 
oversight  and  to  help  it  better  manage  the  development,  such  as  acting  on 
several  recommendations  identified  in  an  Independent  Program 
Assessment  to  address  weaknesses  in  management  responsibility, 
accountability,  and  organizational  structure,  and  establishing  a  dedicated 
execution  team  with  a  focus  on  managing  the  first  satellite  effort. 

DOD  has  estimated  that  the  SBIRS  program  will  be  delayed  by  15  months 
and  cost  $414  million  in  funding  to  resolve  the  flight  software  problems, 
but  these  estimates  appear  too  optimistic.  For  example,  the  productivity 
estimates  that  are  based  on  the  program’s  ability  to  develop,  integrate,  and 
test  software  in  time  to  meet  the  schedule  have  been  assessed  as  low — by 
technical  contractors — ranging  from  5  to  60  percent  in  confidence  for 
meeting  the  schedule  goal.  Further,  the  review  teams  who  approved  the 
designs  to  start  coding  software  report  that  the  program’s  aggressive 
schedule  is  a  major  challenge  because  it  allows  “little  margin  for  error.”  In 
addition,  DOD  has  introduced  program  risk  by  requesting  and  receiving 
waivers  for  the  purpose  of  streamlining  important  software  development 
processes  to  meet  the  aggressive  schedule.  The  waivers  will  allow  the 
program  to  deviate  from  disciplined  processes  in  order  to  compress  the 
schedule  and  meet  the  goal.  Finally,  some  program  elements  are  already 
behind  schedule,  and  thousands  of  software  activities  and  deliverables 
remain  that  must  be  integrated  without  significant  consequence  across  the 
broad  spectrum  of  development  elements,  such  as  integration  with 
ground,  space,  and  database  systems.  Also,  the  launch  range  needed  by 
SBIRS  to  launch  the  first  satellite  is  scheduled  for  use  by  other  satellite 
programs  prior  to  SBIRS.  Delay  in  these  other  satellite  programs  could 
create  unintended  consequences.  Should  DOD  need  additional  time  or 
encounter  problems  beyond  what  was  marginally  planned  for,  more  funds 
will  be  needed  and  launch  of  the  satellite  in  December  2009  could  be  in 
jeopardy. 

We  are  making  recommendations  to  the  Secretary  of  Defense  regarding 
the  development  of  realistic  cost  and  schedule  estimates  commensurate 
with  acceptable  program  risk  to  increase  the  confidence  of  success,  and 
adherence  to  disciplined  software  practices.  DOD  partially  concurred  with 
our  recommendation  to  revise  the  cost  and  schedule  estimates  based  on 
more  realistic  assumptions,  and  concurred  with  our  recommendation  to 
require  the  contractor  to  make  adherence  to  disciplined  practices  a 
priority.  On  the  recommendation  to  develop  realistic  cost  and  schedule 
estimates,  DOD  stated  that  the  current  goals  are  executable  on  the  basis  of 
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available  management  reserve  and  schedule  margin,  as  well  as  additional 
funds  that  have  been  approved  by  Congress  in  the  event  of  any 
unforeseeable  problems  or  delays.  DOD  further  stated  it  would  consider 
modifying  the  cost  and  schedule  goals  based  on  the  results  of  an  ongoing 
flight  software  assessment.  While  DOD’s  plan  to  assess  software  and  its 
willingness  to  revise  the  cost  and  schedule  goals  appear  plausible,  we 
believe  this  approach  falls  well  short  of  a  more  reasonable  approach  to 
revise  the  estimates  based  on  realistic  assumptions  to  increase  the 
confidence  of  success.  In  light  of  the  program’s  risks,  poor  performance 
history,  and  technical  challenges  expected  during  integration,  we  maintain 
that  developing  goals  based  on  realistic  assumptions  would  place  DOD  in 
a  position  to  achieve  cost  and  schedule  goals  with  greater  confidence. 


Background 


DOD  initiated  the  SBIRS  program  to  meet  all  military  infrared  surveillance 
requirements  through  a  single,  integrated  system,  and  to  provide  better 
and  timelier  data  to  the  Unified  Combatant  Commanders,  U.S.  deployed 
forces,  U.S.  military  strategists,  and  U.S.  allies.  SBIRS  is  to  replace  the 
existing  infrared  system,  the  Defense  Support  Program,  which  has 
provided  early  missile  warning  information  since  the  1970s.  The  SBIRS 
program  was  originally  conceived  as  having  high-  and  low-orbiting  space- 
based  components  and  a  ground  segment  for  mission-data  processing  and 
control  to  improve  current  capabilities.  In  2001,  the  SBIRS  Low 
component  was  transferred  from  the  Air  Force  to  the  Missile  Defense 
Agency  and  renamed  the  Space  Tracking  and  Surveillance  System.  The  Air 
Force  continued  developing  SBIRS  High  (herein  referred  to  as  “SBIRS”). 

It,  along  with  its  associated  ground  segment,  is  one  of  DOD’s  highest 
priority  space  programs. 

The  SBIRS  program  originally  consisted  of  four  satellites  to  operate  in 
geosynchronous  earth  orbit  (GEO),  plus  one  spare,  an  infrared  sensor 
placed  on  two  host  satellites  in  highly  elliptical  orbit  (HEO) — known  as 
“HEO  sensors” — and  a  ground  segment  for  mission-data  processing  and 
control. 

The  SBIRS  GEO  satellite  is  designed  to  support  two  infrared  sensors — a 
scanning  sensor  and  a  staring  sensor.  The  first  GEO  satellite  is  commonly 
referred  to  as  GEO  1.  Figure  1  shows  the  GEO  satellite  that  is  to  operate  in 
space. 
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Figure  1:  SBIRS  Satellite 


Source:  Lockheed  Martin  Space  Systems  Company,  Sunnyvale,  California.  ©  2007  Lockheed  Martin  Corporation. 


As  a  result  of  past  technical  and  program  difficulties  experienced  during 
sensor  and  satellite  development,  the  SBIRS  program  has  encountered 
cost  and  schedule  increases.  These  difficulties  have  led  DOD  to 
restructure  the  program  multiple  times,  including  revising  program  goals 
in  2002,  2004,  and  2006.  For  example,  in  2002,  the  program  faced  serious 
problems  with  software  and  hardware  design  progress  and,  in  the 
Conference  Report  accompanying  the  National  Defense  Authorization  Act 
for  Fiscal  Year  2002,  conferees  recommended  cutting  advance 
procurement  funding  due  to  concerns  about  program  developments  and 
the  unclear  status  of  the  SBIRS  program.  At  that  time,  the  first  satellite 
launch  slipped  from  2002  to  2006.  In  late  2005,  SBIRS  was  restructured  for 
a  third  time  which  stemmed  from  a  160  percent  increase  in  estimated  unit 
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cost,  triggering  a  fourth  Nunn-McCurdy^  breach,  which  again  postponed 
the  delivery  of  promised  capabilities  to  the  warfighter. 


Flight  Software  The  flight  system  software  is  expected  to  control  the  GEO  satellite’s 

mission  critical  functions  and  activities.  Unlike  other  software  programs 
that  can  be  deferred  and  uploaded  to  the  satellite  after  launch,  the  flight 
software  cannot  be  deferred  because  it  is  critical  to  the  satellite’s 
operation  and  function.  The  flight  software  is  expected  to  operate,  control, 
and  monitor  the  GEO  satellite’s  health,  status,  and  safety.  Based  on  the 
original  design,  the  flight  software  was  to  operate  on  two  of  four  computer 
processors  onboard  the  satellite  and  perform  important  fimctions  and 
operations,  such  as  telemetry,  thermal  control,  power  management,  and 
fault  detection  activities.'*  Figure  2  shows  a  simplified  diagram  of  the 
original  flight  software  design. 


^10  U.S.C.  §  2433,  commonly  known  as  “Nunn-McCurdy,”  generally  requires  DOD  to  review 
programs  and  report  to  Congress  whenever  certain  unit  cost  growth  thresholds  are 
reached. 

^Satellites  primarily  consist  of  the  payload  and  the  bus.  Currently,  DOD’s  buses  are  custom- 
made  for  each  space  program. 
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Source:  Lockheed  Martin  (data);  GAO  (analysis  and  presentation). 


Origin  and  Chronology  of  in  1996,  development  of  the  flight  software  began  as  an  independent 
Flight  Software  Events  research  and  development  project  by  Lockheed  Martin — referred  to  as 

reusable  flight  software  (RFSW) — to  be  used  for  multifunctional  “bus” 
purposes.*^  In  2004,  the  RFSW  was  provided  to  the  SBIRS  program  for 
development  as  the  flight  system  software  to  operate,  control,  and  monitor 
the  GEO  satellite’s  health,  status,  and  safety.  At  that  time,  the  software 
needed  to  address  1261  requirements  in  order  to  satisfy  the  specific  flight 
software  system  needs  for  the  GEO  satellite.  From  2005  to  2006,  the  Air 
Force  and  Lockheed  Martin  conducted  detailed  requirements  reviews  that 
resulted  in  the  delivery  of  flight  software  that  was  integrated  into  the 
satellite’s  computers. 

In  January  2007,  the  flight  software  underwent  testing  in  a  space 
representative  environment  called  thermal  vacuum  testing  and 
experienced  a  higher  number  of  unexpected  and  unexplained  failures.  By 
April  2007,  in  additional  tests,  the  number  of  problems  escalated  well 


rihe  bus  is  the  platform  that  provides  the  power,  attitude,  temperature  control,  and  other 
support  to  the  satellite  in  space. 
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beyond  what  was  expected.  At  this  time,  Lockheed  Martin  notified  DOD  of 
the  seriousness  of  the  problem.  From  April  2007  to  July  2007,  the  Air  Force 
and  Lockheed  Martin  analyzed  the  problems  and  developed  two  options: 

•  modify  the  existing  software  or 

•  redesign  the  software  by  simplifying  the  architecture,  developing  more 
software,  and  increasing  the  robustness  of  the  fault  management 
system. 

The  Air  Force  chose  to  redesign  the  software  architecture  and  began  its 
work  with  Lockheed  Martin  on  detailed  software  designs  from  September 
2007  to  December  2007.  In  March  2008,  the  new  design  underwent 
Incremental  Design  Review  Block  1  and  was  approved  by  the  program 
review  board  for  the  revised  cost  and  schedule  baseline.  In  April  2008,  six 
independent  review  teams  examined  the  Block  2  design  during  the 
Systems  Engineering  &  Incremental  Design  Review  and  authorized  the  Air 
Force  and  Lockheed  to  proceed  with  formal  software  coding  under  the 
redesign.® 


DOD  Is  Taking  Steps 
to  Mitigate  Software 
Problems,  Including 
Initiatives  to  Improve 
Program  Oversight 


To  mitigate  the  software  problems,  DOD  has  assessed  various  alternatives 
and  developed  an  approach  for  implementing  the  software  redesign  effort 
and  overseeing  its  development.  DOD  and  the  SBIRS  contractor  are  taking 
steps  to  address  problems,  among  others,  with  the  original  software 
architecture.  DOD  has  redesigned  the  architecture,  and  is  in  the  midst  of 
developing  additional  software,  and  testing  elements  critical  to  the 
integration  and  test  of  systems.  DOD  has  also  undertaken  several 
initiatives  to  improve  its  program  oversight  and  to  help  it  better  manage 
the  development,  including  addressing  weaknesses  in  program 
management  responsibility,  accountability,  and  other  areas. 


®FSS  vl.5  Block  2  Systems  Engineering  &  Incremental  Design  Review,  Lockheed  Martin 
Space  Systems  Company,  Sunnyvale,  California. 
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Steps  Have  Been 
Undertaken  to  Address 
Poor  Software 
Architecture 


Major  Redesign  Approved  for 
Coding  Software 


To  address  the  software’s  poor  architectural  design  that  ultimately 
resulted  in  the  unexpected  loss  of  telemetry  and  commanding  for 
extended  periods  and  unexpected  hardware  errors,  a  trade  study  was 
conducted  by  Lockheed  Martin  to  examine  options  for  redesign.  Table  1 
shows  the  trade  study  options  considered,  and  recommendations  made. 


Table  1:  Trade  Study  Options  and  Recommendations  on  Software  Architecture 

Option 

Recommendation 

Distributed  applications  (synchronous) 

Not  recommended  due  to  complexity  and  risk 

Distributed  applications  (asynchronous) 

Not  recommended  due  to  complexity  and  risk; 
has  the  highest  impact  to  ground  systems 

All  applications  on  processor  “B” 

Not  recommended  due  to  complexity  and  risk 

All  applications  on  processor  “A” 

Recommended  as  best  fit  with  component  and 
fault  management  system  designs 

Source:  Lockheed  Martin  (data);  GAO  (analysis  and  presentation). 


As  indicated  in  table  1,  the  trade  study  recommended  a  simplified 
architecture  that  places  all  the  software  applications  on  a  single  processor, 
processor  “A”,  rather  than  using  distributed  applications  because  it 
represents  the  best  fit  with  system  designs.  Lockheed  Martin  officials 
stated  that  the  simplified  software  architecture  will  address  a  number  of 
areas  that  were  problematic  with  the  original  design,  such  as  the  timing  of 
stored  programs  that  failed  during  thermal  vacuum  tests.  Among  other 
elements,  the  new  design  will  involve  the  development  of  additional 
software  that  will  also  increase  the  robustness  of  the  fault  management 
system. 

Approved  in  April  2008,  the  new  designs  have  undergone  numerous 
reviews,  the  last  of  which  was  subjected  to  comprehensive  and  detailed 
examination  involving  six  independent  review  teams.  Teams  comprised  of 
personnel — from  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics;  Aerospace  Corporation,  a  federally 
funded  research  and  development  center  (FFRDC)^;  Lockheed  Martin 


^FFRDCs  are  unique  independent  nonprofit  entities  sponsored  and  funded  by  the. 
government  to  meet  specific  long-term  technical  needs  that  cannot  be  met  by  existing  in 
house  or  contractor  resources.  The  Aerospace  Corporation’s  FFRDC  is  sponsored  by  the 
Air  Force,  and  provides  objective  technical  analyses  and  assessments  for  space  programs 
that  serve  the  national  interest.  As  the  FFRDC  for  nation-security  space,  Aerospace 
supports  long-term  planning  and  the  immediate  needs  of  our  nation’s  military  and 
reconnaissance  space  programs. 
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Progress  Is  Being  Made  to 
Develop  Software  and  Conduct 
Tests 


Corporate;  Air  Force  Space  and  Missiles  Systems  Center  Wing;  and  the 
Software  Engineering  Institute® — evaluated  the  technical  solutions, 
development  approach,  and  readiness  of  the  test  facilities,  among  other 
elements. 

The  objective  of  the  design  review  was  to  authorize  the  start  of  formal 
software  coding.  For  the  incremental  design  review,  independent  review 
teams  were  provided  detailed  information  about  software  issues  on  the 
original  design,  including  the  severity  of  the  issues  and  the  status  of  each. 
Other  information  included  DOD’s  approach  in  managing  risk,  resolution 
of  critical  issues,  disposition  of  deficiency  reports,  requirements  volatility, 
and  integration  with  ground  systems.  Technical  data  included  diagrams  of 
the  simplified  architecture,  operating  system  interface  design,  and  lines  of 
software  code  that  would  be  impacted  from  earlier  designs.  Other 
information  about  the  software  included  designs  of  subsystems, 
schematics,  integration  and  delivery  schedules,  and  productivity  and  sizing 
estimates. 

DOD  is  making  progress  to  develop  needed  software  and  conduct  tests  of 
elements  that  are  critical  to  the  first  satellite  system,  called  GEO  1.  For 
example,  in  June  2008,  DOD  held  a  design  review  on  software  for  the  fault 
management  system  that  elicited  concurrence  from  external  stakeholders 
to  proceed  with  coding  activities.  At  the  same  time,  they  held  a  space 
technical  interchange  meeting  that  provided  consensus  on  the 
methodology  and  a  plan  for  complete  space  vehicle  testing,  including  the 
flight  software.  In  July  2008,  Lockheed  Martin  delivered  63,000  of  the 
projected  67,000  source  lines  of  code  for  the  space  vehicle  and  ground 
software  integration  effort,  including  a  database  that  provided  data  so  that 
development  efforts  could  continue  on  ground  software  and  testing 
activities. 

According  to  Lockheed  Martin,  software  development  efforts  followed  a 
disciplined  process,  except  in  those  cases  where  waivers  were  requested 
and  granted  by  the  software  engineering  process  group.  Figure  2  shows 
Lockheed  Martin’s  process  for  developing  and  qualifying  flight  software. 


®The  Software  Engineering  Institute  is  a  FFRDC  that  works  closely  with  defense  and 
government  organizations,  industry,  and  academia  to  continuously  improve  software 
intensive  systems.  The  Institute’s  core  purpose  is  to  help  organizations  to  improve  their 
software  engineering  capabilities  and  to  develop  or  acquire  the  right  software,  defect  free, 
within  budget  and  on  time. 
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Figure  3:  Flight  Software  Development  Process 


Specification  complete 
Requirements  aliocated 

Quaiification  plan 
compiete 


Development 


•  Requirement  aiiocated 

*  Quaiification  pian 


•  Code  reviews  compiete 


Engineering  Dry-Runs 


Test  cases  documented 
Test  scripts  vaiidated 
Formal  reviews  complete 
Test  set  verified 
Deficiency  reports  closed 


•  Unit  test  peer  reviewed 

•  Integration  plan 
complete 

•  Test  environment  ready 


Test  build  complete 


Formal  Dry-Runs 


•  Final  test  documents/scripts 

•  Test  set  certified 

•  Launch  software  build 

•  Deficiency  reports  closed 


Qualification 


Run  For  Record 


•  Test  exit  report  complete 


Source:  Lockheed  Martin  (data);  GAO  (analysis  and  presentation). 


Risks  Reduced  by  Funding  DOD  has  taken  steps  to  fund  critical  test  bed  resources  that  are  needed  to 

Additional  Test  Resources  adequately  test,  model,  analyze,  and  simulate  software  functions  as  a 

means  to  reduce  integration  and  test  risks,  in  response  to  lessons  learned 
from  the  failed  software  that  identified  the  need  to  add  and  upgrade  their 
simulation  and  test  bed  resomces.  For  example,  an  evaluation  of  the 
software  problems  found  several  contributory  factors  that  prevented  them 
from  identifying  the  software  problems  earlier.  These  include: 
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•  test  beds  that  had  matured  in  parallel  with  the  flight  software  and 
hardware,  making  it  difficult  to  distinguish  between  test  bed  and 
software  issues; 

•  oversubscription  of  test  beds  and  lack  of  simulation  resources  that 
precluded  them  from  checking  out  high-risk  areas  (timing,  and  stored 
programs);  and 

•  insufficient  modeling  of  timing,  and  analysis  of  stored  program 
implementation,  which  might  have  shed  light  earlier  on  lack  of 
robustness. 

In  May  2008,  the  additional  test  bed  and  simulator  was  brought  online  and 
is  currently  in  use. 


Actions  Have  Been 
Undertaken  to  Address 
Program  Weaknesses,  and 
Improve  Oversight  of  GEO 
Development 


DOD  and  Lockheed  Martin  have  undertaken  several  initiatives  to  address 
areas  of  program  risk,  such  as  efforts  to  improve  oversight  of  GEO  1  and 
flight  software  development.  These  include  acting  on  recommendations 
made  in  an  Independent  Program  Assessment  (IPA)  that  was  conducted  to 
ensure  the  validity  of  the  technical,  cost,  and  schedule  baselines.  As  part 
of  the  assessment,  the  IPA  study  assessed  contractor  performance, 
evaluated  program  risk  areas,  and  made  recommendations  on  where 
program  improvements  could  be  made.  In  November  2007,  officials  from 
the  Air  Force,  Lockheed  Martin,  and  Aerospace  Corporation  reported  the 
IPA  findings.  Table  2  shows  the  IPA  findings,  recommendations,  and  status 
of  implementation  efforts. 


Table  2:  IPA  Findings,  Recommendations,  and  Status  of  Implementation 

Finding 

Recommendation 

Implemented? 

(as  of  April  2008) 

1 .  Lockheed  Martin’s  program  process 
discipline  is  poor 

•  Engage  Lockheed  Martin  functional  areas  and  ensure  that 
processes  are  being  followed 

Yes 

2.  Air  Force  has  limited  management  control 
over  SBIRS 

•  Amend  contract  to  provide  necessary  management  control 

Yes 

3.  Adversarial  relationships  exist  between  Air 
Force  and  Lockheed  Martin 

•  Fix  responsibility,  accountability,  and  authority  disconnects 

Yes 

4.  Government  organizational  structure  Is 
flawed  because  cost  and  schedule 
responsibilities  are  separated. 

•  Combine  in  a  single  office  the  review  of  contractor  cost  and 
schedule  data 

Yes 

5.  Focal  point  for  FSS  completion  is  needed 

•  Designate  a  program  manager  within  flight  software  system 

•  Establish  giver/receiver  relationships 

Yes 

Source:  Aerospace  Corporation  (data)  and  U.S.  Air  Force  (data);  GAO  (analysis  and  presentation). 
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As  indicated  in  table  2,  the  Air  Force  and  Lockheed  Martin  have  taken 
actions  to  address  areas  of  risk.  Among  others,  these  actions  included 
deliberately  emphasizing  the  software  development  process  where 
adherence  to  process  disciplines  was  lacking,  and  enhancing  the 
interaction  between  cost  and  schedule  functions  where  the  Air  Force 
organization  structure  was  found  to  be  flawed  because  it  did  not  mirror 
the  contractor’s  more  traditional  approach  where  these  functions  are 
combined  for  better  program  control. 

To  improve  the  oversight  and  management  of  the  GEO  1  satellite  and 
software  development,  the  Air  Force  and  Lockheed  Martin  established  a 
dedicated  execution  team  with  a  focus  on  overseeing  the  test,  integration, 
and  assembly  of  software  and  hardware,  and  ensuring  delivery  of  the  GEO 
1  satellite.  The  execution  team  is  a  joint  effort  that  includes  the  Air  Force, 
Lockheed  Martin,  and  Aerospace  Corporation.  As  part  of  the  management 
approach,  the  execution  team  is  responsible  for  conducting  daily  meetings 
to  review  “inch  stone”  metrics  and  to  resolve  issues.  The  execution  team 
also  meets  weekly  with  the  Executive  Program  Management  leadership  to 
provide  early  insight  on  issues  and  resolve  organizational  weaknesses,  and 
conduct  monthly  reviews  with  senior  executives  to  provide  consistent 
communication  and  allow  opportunity  for  guidance.  According  to  DOD 
officials,  the  execution  team  not  only  improved  oversight  of  software 
development  and  management  of  the  GEO  1  effort,  but  also  addressed 
weaknesses  identified  in  the  IPA  study.  For  example,  these  weaknesses 
included,  among  others,  the  need  to  fix  the  program’s  responsibility, 
accountability,  and  authority  disconnects.  Officials  reported  that  the 
execution  team  helped  alleviate  the  strained  relationships  that  had  existed 
between  the  Air  Force  and  Lockheed  Martin  where  adversarial 
relationships  and  morale  problems  were  evident. 


Page  13 


GAO-08-1073  SBIRS  Software 


DOD’s  Plan  for 
Resolving  the 
Software  Problem  Is 
Optimistic 


While  DOD  has  estimated  that  the  SBIRS  program  will  be  delayed  by  15 
months  and  cost  $414  million  to  resolve  the  software  problems,  those 
estimates  appear  too  optimistic,  given  the  cost  and  schedule  risks 
involved.  For  example,  SBIRS  contractors’  report  low  confidence  that 
software  can  be  produced  in  time  to  meet  the  December  2009  satellite 
launch  goal.  Further,  DOD  and  the  contractor  face  significant  challenges 
and  risks  that  could  result  in  more  time  and  money  being  required  to  meet 
program  goals,  to  include  the  bypassing  of  some  disciplined  software 
practices  that  add  risk  to  cost  and  schedule.  Finally,  as  of  August  2008, 
DOD  reported  that  SBIRS  was  already  behind  schedule  on  some  software 
development  efforts,  and  thousands  of  activities  remain  that  must  be 
integrated  and  tested  across  various  systems,  with  cost  and  schedule 
implications,  if  problems  or  unintended  consequences  occur. 


Low  Confidence  That  A  major  concern  is  the  infeasibility  of  producing  the  software  in  time  to 

Software  Can  Be  Produced  nieet  the  estimated  launch  goal.  For  example,  technical  contractors — 
to  Meet  Cost  and  Schedule  Aerospace  Corporation,  Galorath  Inc.,  and  Lockheed  Martin — estimated 
Goals  confidence  to  be  “low”  that  software  can  be  developed  within  the  tight 

time  frames.  These  estimates  are  based  on  widely  accepted  models 
(System  Evaluation  and  Estimation  of  Resources,  Software  Estimating 
Model,  and  Risk  Assessment)  that  take  into  account  the  effective  size  of 
the  software,  staffing  of  the  effort,  complexity,  volatility  of  software 
requirements,  and  integration  and  risk  of  anticipated  rework  and  failure  in 
system  tests.  Using  DOD’s  self-imposed  baseline  schedule  goal,  software 
productivity  estimates  show  very  low  confidence  levels  that  the  schedule 
goal  can  be  met.  Table  3  shows  the  confidence  in  meeting  the  GEO  1 
launch  goal  in  December  2009  (various  models  used). 


Table  3:  Confidence  Level  to  Produce  Software  to  Meet  GEO  1  Schedule 

Confidence  level 

Contractors 

Estimated  launch  goal 

Less  than  10  percent 

Aerospace  Corporation 

December  2009 

5  percent 

Galorath,  Inc. 

December  2009 

50  percent 

Lockheed  Martin 

December  2009 

Source:  U.S.  Air  Force  (data);  GAO  (analysis  and  presentation). 


As  indicated  in  table  3,  one  estimate  shows  only  a  5  percent  confidence 
that  the  software  can  be  produced  in  time  to  meet  the  schedule  goal,  while 
the  other  estimate  shows  a  less  than  10  percent  confidence  level. 
Lockheed’s  own  software  productivity  estimate  shows  a  50  percent 
confidence  level  in  meeting  the  December  2009  launch  schedule,  but  its 
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estimate  assumes  (1)  a  higher  productivity  than  has  been  demonstrated, 
and  (2)  the  software  will  require  less  effort,  which  has  not  been  the 
program’s  experience.  According  to  DOD’s  Cost  Analysis  Improvement 
Group,  if  productivity  on  software  does  not  materialize,  or  problems  occur 
during  testing  and  integration  beyond  what  was  marginally  planned  for, 
then  it  could  cost  an  additional  $400  million  for  each  year  of  schedule 
slippage. 


Major  Challenge  and  Risks 
to  the  Redesign  and 
Development  Effort  Still 
Exist 


Based  on  an  April  2008  review  of  the  revised  software  designs  and 
software  development  approach,  the  independent  review  teams — 
comprised  of  persormel  from  the  Office  of  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology,  and  Logistics;  Aerospace  Corporation; 
Lockheed  Martin  Corporate;  Air  Force  Space  and  Missiles  Systems  Center 
Wing;  and  the  Software  Engineering  Institute — concluded  that  the 
program  should  proceed  with  formal  software  coding,  but  also  expressed 
concern  about  the  ambitious  schedule.  Specifically,  the  review  teams  cited 
the  program’s  aggressive  schedule  as  a  major  challenge  because  it  allows 
“little  margin  for  error”  and  concluded  the  program  faces  high  risk  of  not 
meeting  the  schedule.  Table  4  shows  the  weaknesses  and  risks  to  software 
development. 


Table  4:  Weaknesses  and  Risks  to  Software  Development 

Weaknesses 

• 

Schedule  pressure,  and  alignment  of  code  and  designs 

• 

Code  complexity  impacting  unit  testing 

• 

Late  integration  with  ground  software 

• 

Significant  amount  of  work  remaining 

Risks 

• 

Concurrent  systems  engineering  and  software  development 

• 

Code  development  requiring  more  labor  than  estimated 

• 

Additional  system  or  software  testing  required  beyond  plans 

• 

Qualification  of  test  products  behind  schedule 

• 

Systems  engineering  completion  may  require  more  effort 

Source:  Lockheed  Martin  (data);  GAO  (analysis  and  presentation). 


Although  the  Air  Force  and  Lockheed  Martin  are  committed  to  the  effort 
and  have  built  in  a  120-day  margin  to  fix  unexpected  and  unforeseeable 
problems,  a  computer  engineer  from  the  Defense  Contract  Management 
Agency  who  is  familiar  with  the  program  believes  that  the  margin  is 
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insufficient  because  the  planned  schedule  considers  only  routine 
development  activities,  and  that  additional  time  will  likely  be  needed  to 
address  any  unanticipated  problems. 

Bypassing  Disciplined  Software  Further,  to  meet  the  cost  and  schedule  goals,  the  program  is  using 

Practices  Adds  Risk  approaches  that  will  increase  program  risk.  These  risks  stem  from 

waivers,  which  were  requested  by  Lockheed  Martin,  as  specified  by 
software  provisions  in  the  program’s  software  development  process.  In 
following  the  SBIRS  Software  Development  Plan,  for  Flight  Software 
System  1.5,  waivers  were  generated  and  approved  by  a  software 
engineering  process  group  so  that  developers  could  deviate  from  the 
established  processes.  These  deviations  from  the  disciplined  development 
process  allowed  the  program  to  shortcut  important  processes  in  order  to 
meet  the  ambitious  schedule  goal,  rather  than  follow  a  disciplined  process 
to  develop  software.  For  example,  a  waiver  was  granted  for  software 
design  to  be  done  in  parallel  with  the  software  specification  activity. 
However,  according  to  DOD,  the  risk  is  that  requirements  could  be 
rejected  and  that  rework  may  be  required  in  coding  or  design.  Another 
waiver  was  granted  for  software  unit  integration  testing  to  be  done  in 
parallel  with  formal  unit  testing.  According  to  DOD,  the  risk  is  that  formal 
unit  testing  may  find  problems  that  were  not  identified  during  prior 
informal  (developer)  unit  testing,  thereby  necessitating  possible  rework. 


Cost  and  Schedule  Goals 
Are  at  Risk  Because  Some 
Software  Elements  Are 
Behind  Schedule,  and 
Complex  Integration  and 
Other  Activities  Remain 


Some  of  the  flight  software’s  elements  are  already  behind  schedule  and  a 
significant  amount  of  activities  remain  to  be  done,  posing  concern  to  DOD. 
For  example,  DOD  reported  that,  as  of  August  2008,  the  software 
qualification  test  case  and  script  development  effort  was  already  a  month 
behind  schedule.  Also,  final  delivery  of  the  Block  2  flight  software  is  now 
forecasted  to  be  at  least  2  weeks  late.  Other  problems  that  could  set  back 
SBIRS  are  the  thousands  of  integration  and  coordination  activities  that 
must  take  place  as  they  ramp  up.  For  example,  Lockheed  Martin  reports 
that  the  schedule  has  more  than  14,500  tasks  that  will  occur,  beginning  in 
January  2008,  across  multiple  systems.  This  means  that  the  flight  software 
test  activities  and  integration  efforts  must  all  be  integrated  in  a  “single¬ 
flow”  without  consequence  across  a  broad  spectrum  of  systems,  such  as 
integration  with  ground,  space,  and  database  systems,  among  others. 
Software  experts,  independent  reviewers,  and  government  officials 
acknowledged  that  the  aggressive  schedule,  when  combined  with  the 
significant  amount  of  work  that  remains,  is  the  biggest  challenge  facing  the 
program. 
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Still,  there  are  external  factors  that  could  create  schedule  impacts  for 
meeting  the  SBIRS  schedule  goal.  For  example,  DOD  reports  that  the  GEO 
1  satellite  launch  could  be  affected  by  other  satellites  scheduled  to  launch 
prior  to  the  SBIRS  launch.  Essentially,  these  launch  activities  use  the  same 
launch  range  resources  that  will  be  required  to  launch  the  GEO  1  satellite, 
and  delays  in  any  of  these  events  could  create  unintended  consequences  to 
the  SBIRS  GEO  1  launch  goal. 


Conclusions 


Given  the  technical  complexity  of  the  program  and  SBIRS’  poor  program 
history,  it  is  unwise  for  DOD  to  pursue  such  ambitious  goals  for  resolving 
the  flight  software  problem.  More  than  12  years  after  its  inception,  the 
SBIRS  program  continues  to  face  major  challenges  that  have  proven 
technically  challenging  and  substantially  more  costly  than  originally 
envisioned.  The  testing  failure  of  the  flight  software  is  further  proof  that 
sophisticated  technology  and  inherent  complexities  related  to  software 
continue  to  be  underestimated.  To  its  credit,  DOD  has  instilled  greater 
discipline  by  involving  outside  experts,  regaining  control  of  development 
activities,  and  dealing  with  the  poor  relationships  that  had  existed  for 
some  time.  To  ensure  that  such  steps  can  lead  to  success,  adherence  to 
disciplined  software  practices  should  be  made  a  priority  over  steps  or 
measures  taken  to  compress  the  schedule  for  the  sake  of  meeting  the  self- 
imposed  launch  goal.  Prioritizing  such  disciplines  will  improve  efforts  to 
acquire  a  better  product,  increase  executability  of  the  program,  and 
reduce  program  risk.  In  turn,  establishing  goals  that  are  synchronized  with 
such  priorities  will  allow  DOD  to  achieve  expectations  and  program 
deliverables  with  greater  reliability.  Essentially,  these  will  position  the 
leadership  to  better  direct  investments  by  establishing  goals  with  greater 
confidence  that  they  can  be  achieved. 


Recommendations 
Executive  Action 


To  better  ensure  that  SBIRS  can  meet  the  cost  and  schedule  goals  for 
resolving  the  flight  software  problems  as  well  as  launch  the  first  satellite 
on  schedule,  we  recommend  that  the  Secretary  of  Defense 


•  revise  the  cost  and  schedule  estimates  based  on  more  realistic 
assumptions  to  increase  the  confidence  of  success,  and 

•  require  that  the  contractor  make  adherence  to  disciplined  software 
practices  a  priority  to  reduce  program  risk. 
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Agency  Comments 
and  Our  Evaluation 


DOD  provided  us  with  written  comments  on  a  draft  of  this  report.  DOD 
partially  concurred  with  our  recommendation  to  revise  the  cost  and 
schedule  estimates  based  on  more  realistic  assumptions,  and  concurred 
with  our  recommendation  to  require  the  contractor  to  make  adherence  to 
disciplined  practices  a  priority.  DOD’s  comments  appear  in  appendix  11. 

In  its  comments,  DOD  partially  concurred  with  the  recommendation  that 
the  cost  and  schedule  estimates  be  revised  based  on  more  realistic 
assumptions  to  increase  the  confidence  of  success.  DOD  noted  that  the 
current  goals  are  executable  on  the  basis  of  available  management  reserve 
and  schedule  margin.  In  the  event  that  the  program  encounters  any 
unforeseeable  problems  that  may  cause  further  delays,  DOD  stated  that 
Congress  has  approved  an  additional  $45  million  in  funding  to  mitigate  any 
future  launch  delays.  The  department  pointed  out  that  OSD  is  working 
with  the  SBIRS  program  to  hold  a  more  specific  review  of  the  flight 
software.  Based  on  the  results  of  this  review,  DOD  stated  it  would 
consider  them  in  any  decision  to  modify  the  cost  and  schedule  estimates. 
DOD  expects  these  assessments  to  be  complete  by  the  end  of  the  2008 
calendar  year. 

As  indicated  in  our  report,  SBIRS  has  been  restructured  several  times 
because  it  underestimated  the  technical  complexity  and  inherent 
challenges  associated  with  software,  among  other  technical  elements. 
Neither  the  software  assessment  conducted  to  determine  the  confidence 
of  producing  software  nor  the  independent  reviewers  who  examined  the 
redesign  approach  indicated  that  the  current  goals  were  executable. 
Rather,  as  we  noted,  software  experts,  independent  reviewers,  as  well  as 
the  government  officials  we  interviewed  expressed  concern  over  the 
aggressive  schedule  and  questionable  schedule  margin,  which  the  Defense 
Contract  Management  Agency  believes  is  insufficient.  Moreover,  as  we 
previously  reported  and  noted  in  this  report,  the  expenditure  of 
management  reserves  has  been  particularly  problematic  because  these 
funds  were  being  rapidly  spent.  Further,  while  OSD’s  plan  to  assess 
software  and  its  willingness  to  revise  the  cost  and  schedule  goals  appear 
plausible,  we  believe  this  approach  falls  well  short  of  a  more  reasonable 
approach  to  increase  the  confidence  of  success  for  the  reasons  we  cited. 

In  light  of  the  program’s  risks,  poor  performance  history,  and  technical 
challenges  expected  during  integration,  we  maintain  that  establishing 
goals  that  are  based  on  more  realistic  assumptions  would  place  DOD  in  a 
better  position  to  achieve  cost  and  schedule  goals  with  greater  confidence. 

DOD  concurred  with  the  second  recommendation  stating  that  adherence 
to  disciplined  software  development  processes  improves  the  quality  and 
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predictability  of  the  software  development  while  reducing  the  amount  of 
rework.  DOD  further  states  that  the  program  office  and  the  contractor 
jointly  accepted  two  process  waivers  to  streamline  the  process,  but  that 
these  waivers  have  had  no  adverse  impact  on  the  software  development 
effort.  In  order  to  keep  the  focus  on  quality  software  deliveries,  DOD 
noted  that  the  program  would  disapprove  any  waivers  which  might 
compromise  the  team’s  ability  to  complete  the  development. 

We  are  encouraged  by  DOD’s  efforts  to  adhere  to  disciplined  software 
processes  to  improve  the  quality  and  predictability  of  development.  In  this 
endeavor,  DOD  states  that  it  would  disapprove  any  waivers  that  could 
compromise  the  development  effort.  However,  it  is  unclear  exactly  what 
criteria  DOD  will  use  to  determine  whether  a  waiver  will  compromise 
development  efforts.  Without  this,  there  is  no  mechanism  to  ensure  that 
any  waivers  that  are  granted  will  not  have  a  material  effect  on  software 
development. 

We  also  received  technical  comments  from  DOD  which  have  been 
addressed  in  the  report,  as  appropriate. 


We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense;  the  Office 
of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics;  the  Secretary  of  the  Air  Force;  and  the  Director,  Office  of 
Management  and  Budget.  Copies  will  also  be  made  available  to  others  on 
request.  In  addition,  the  report  will  be  made  available  at  no  charge  on  the 
GAO  Web  site  at  http://www.gao.gov. 

If  you,  or  your  staff,  have  any  questions  concerning  this  report,  please 
contact  me  at  (202)  512-4589.  Contact  points  for  our  offices  of 
Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
of  this  report.  The  major  contributors  are  listed  in  appendix  III. 


Cristina  T.  Chaplain 
Director 

Acquisition  and  Sourcing  Management 
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List  of  Congressional  Committees 

The  Honorable  Bill  Nelson 
Chairman 

The  Honorable  Jeff  Sessions 
Ranking  Member 
Strategic  Forces  Subcommittee 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Ellen  Tauscher 
Chairwoman 

The  Honorable  Terry  Everett 
Ranking  Member 
Strategic  Forces  Subcommittee 
Committee  on  Armed  Services 
House  of  Representatives 
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Appendix  I:  Scope  and  Methodology 


To  identify  the  Space  Based  Infrared  System’s  (SBIRS)  approach  to  mitigate 
the  flight  software  problems,  we  reviewed  the  plans  and  alternatives  the 
Department  of  Defense  (DOD)  put  in  place  to  mitigate  the  software 
problem.  We  also  interviewed  Air  Force,  Defense  Contract  Management 
Agency,  and  Lockheed  Martin  officials  who  were  responsible  for 
management  and  oversight  of  the  software  development  effort.  We  also 
examined  technical  reports,  studies,  and  analyses  about  the  factors  that 
contributed  to  the  flight  software  problems,  as  well  as  planning  documents 
and  alternatives  that  were  considered  in  fixing  the  software  problem. 

To  assess  the  cost  and  schedule  risks  and  challenges  of  the  way  forward,  we 
held  discussions  with  both  the  DOD  and  Lockheed  Martin  on  their  efforts  to 
assess  the  program  risks  and  challenges,  including  their  approach  to 
manage,  mitigate,  and  redesign  the  flight  software  that  is  to  operate,  control 
and  monitor  the  satellite’s  health,  status,  and  safety.  We  also  reviewed 
schedules,  risk  reports,  analyses,  program  assessments,  and  independent 
review  reports  pertaining  to  the  flight  software’s  redesign,  and  selected 
assessments  by  independent  sources  that  were  used,  in  part,  as  basis  for 
selecting  December  2009  as  the  launch  goal  for  the  GEO  1  satellite.  We  also 
interviewed  Air  Force  and  contractor  officials  responsible  for  developing 
and  executing  the  redesign,  including  a  contractor  hired  for  their  expertise 
in  estimating  software  productivity. 

We  conducted  this  performance  audit  at  the  Office  of  the  Secretary  of 
Defense,  Washington  D.C.;  Space  and  Missile  Systems  Center,  Los  Angeles 
Air  Force  Base,  California;  and  Lockheed  Martin  and  the  Defense  Contract 
Management  Agency,  Sunnyvale,  California  from  April  to  August  2008  in 
accordance  with  generally  accepted  auditing  standards.  Those  standards 
require  that  we  plan  and  perform  the  audit  to  obtain  sufficient,  appropriate 
evidence  to  provide  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives.  In  addition,  we  drew  from  our  body  of  past 
work  on  weapon  systems  acquisition  practices  and  disciplined  software 
practices.  We  believe  that  the  evidence  obtained  provides  a  reasonable 
basis  for  our  findings  and  conclusions  based  on  our  audit  objectives. 
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Appendix  II:  Comments  from  the  Department 
of  Defense 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 
3000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3000 


ACQUISITION 

TECHNOLOGY 

ANDLOGI8T.es  ggp 


Ms.  Cristina  Chaplain 

Director,  Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W, 

Washington,  DC  20548 

Dear  Ms.  Chaplain; 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report  GAO-08- 
1073,  “SPACE  ACQUISITIONS:  DoD’s  Goals  for  Resolving  Space  Based  Infrared  System 
Software  Problems  Are  Ambitious,”  dated  August  22,  2008,  (GAO  Code  120761).  Detailed 
comments  on  the  two  report  recommendations  are  enclosed. 


Space  &  Intelligence  Capabilities 


Enclosure: 
As  stated 
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Appendix  II:  Comments  from  the  Department 
of  Defense 


GAO  Draft  Report  Dated  August  22,  2008 
GAO-08-1073  (GAO  CODE  120761) 


“SPACE  ACQUISITIONS:  DOD’S  GOALS  FOR  RESOLVING  SPACE 
BASED  INFRARED  SYSTEM  SOFTWARE  PROBLEMS  ARE  AMBITIOUS” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary  of  Defense,  revise  the  cost 
and  schedule  estimates  based  on  more  realistic  assumptions  to  increase  the  confidence  of 
success,  (Page  17/GAO  Draft  Report) 

POD  RESPONSE:  Partially  concur.  While  the  current  contractor  cost  and  schedule  baseline  is 
aggressive  and  contains  risks,  we  believe  the  remaining  Flight  Software  Subsystem  (FSS) 
development  is  still  executable  within  the  available  management  reserve  and  schedule  margin. 
Congress  has  approved  an  additional  $45M  in  Omnibus  funding  to  provide  mitigation  against  a 
future  launch  date  delay  as  a  result  of  any  unforeseen  program  problems,  to  include  Flight 
Software  development  and  qualification  delays.  The  Wing  has  recently  completed  an  integrated 
baseline  review  of  the  program.  In  addition,  the  program  office  is  working  with  OSD  to  hold  a 
more  specific  review  of  the  FSS  effort.  These  results  will  be  considered  in  any  decision  to 
modify  FSS  cost  and  schedule  estimates.  These  assessments  will  be  complete  by  the  end  of  the 
calendar  year. 

RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense  require  that  the 
contractor  make  adherence  to  disciplined  software  practices  a  priority  to  reduce  program  risk. 
(Page  1 7/GAO  Draft  Report) 

POD  RESPONSE:  Concur.  The  DoD  agrees  adherence  to  disciplined  software  development 
processes,  as  outlined  in  the  SBIRS  Software  Development  Plan,  improves  the  quality  and 
predictability  of  the  software  development  while  reducing  the  amount  of  rework.  To  date,  the 
program  office  has  jointly,  with  the  contractor,  accepted  the  two  minor  process  waivers 
mentioned  in  the  report  to  streamline  the  development  process  and  reduce  the  schedule  risk 
associated  with  the  December  2009  projected  launch  date.  Those  waivers  have  had  no  adverse 
impacts  to  the  FSS  development.  To  keep  focus  on  quality  software  deliveries  in  support  of 
space  vehicle  testing  and  operations,  the  program  office  will  disapprove  any  waivers  which 
compromise  the  team’s  ability  to  complete  the  development. 
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Appendix  III:  GAO  Contact  and  Staff 
Acknowledgments 


Contact 


Cristina  T.  Chaplain,  (202)  512-4869  or  chaplainc@gao.gov 
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GAO’S  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accoimtability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  pohcies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  rehability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday,  GAO  posts 
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Congressional 

Relations 

Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accoimtability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngcl@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 

PRINTED  RECYCLED  PAPER 

